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Good afternoon. I am honoreo to be here today and look forward to our discussion after my 
presentation. In the time I have I want to talk about three questions: What skills do we need in the 
economy? What are we getting fron, our schools? How do we get what we need? Specifically, I 
want to talk about three issues: (1) changes in the nature and structure of work in the American 
economy; (2) the implications of those changes and of a "gathering storm" of research in die cognitive 
sciences for K-12 problems in what we teach, to whom we teach it, when we teach it, and how we 
teach it; and (3) using substantially changed accountability measures to get what we need from our 
schools. 

I. THE TRANSFORMATION OF THE AMERICAN ECONOMY 

First, what is going on in the American economy that effects the requirements for our schools? 

Two data sources tell us about the skill implications of changes in the American economy: (1) 
data on employment by occupations, past and projected; and (2) industry and occupational case studies 
that reveal qualitative changes in the nature and structure of work that occupational titles conceal. 

SKILL TRENDS IN EMPLOYMENT BY OCCUPATION 

Census and BLS statistics show a clear long-term increase in skill requirements between 1900 
and 1980. Throughout the century the economy has been shedding lower-skilled and adding higher- 
skilled jobs. For example, in 1900 about 30 percent of the labor force woifced as laborers, either farm 
or non-farm; about 10 percent in either professional, technical, or managerial occupations. B; 1980 
the picture had roughly reversed, about 6 percent working as laborers and 26 percent as professionals, 
technicians, or managers. 

This process continued through the 1980s and is projected to contirue through the 1990s. 
When we look at the total labor force for the 1976-1988 period, we find that the higher-skilled 
occupations grew at almost three times the rate of the lower-skilled jcbs. 1 Although the higher-skilled 
occupations still accounted for less than 40 percent of total employment in 1988, they accounted for 



1 We define higher-skill occupations as those with educational levels, measured by the share of 
the workers in the occupation with at least some college, above the average for the labor force as a 
whole. In this definition the higher-skilled occupations include the executive, administrative, and 
managerial; professional; technician; and marketing and sales occupations. The lower-skill jobs 
include administrative support; service; precision production and craft; operator and fabricator, private 
household; and fanning, forestry, and f shing jobs. 



more than one-half of all of the net employment growth between 1976 and 1988. For the period 
1988-2000, the higher-Skilled occupations are expected to grow about twice as fast as the losver-skilled 
occupations. If we use the educational disiribution of the 1988 incumbents of occupations, we find 
that the new jobs expected to be created between 1988 and 2000 will have higher educational levels 
than current jobs. For example, 17 percent of :urrent jobs are held by workers without a high school 
degree. Only 13 percent of the new jobs expected to be created between 1988 and 2000 will go to 
those without a high school degree, even if the 1988 educational distribution of each occupation does 
not change between 1988 and 2000. The discrepancy between current jobs and new jobs is even 
greater for college graduates. In 1988 college graduates held 22 percent of the jobs. However, 30 
percent of the new jobs will go to workers who have college degrees, 

INDUSTRY CASE STUDIES OF CHANGES IN THE NATURE AND STRUCTURE OF WORK 

'Economic restructuring" essentially means a fracturing of old relationships between 
occupational titles, job content, and skill requirements. It is under these conditions that occupational 
and industry case studies become critical complements to BLS statistics. In the context of our 
National Center on Education and Employment research program, Columbia University economists, 
Drs. Thieny Noydle and Thomas Bailey (Noyelle, 1987, 1988; Bailey and Noyelle, 1988; Bailey, 
1988), document these changes in different types of industries: service and manufacturing, top-of-the- 
product cycle and bottom-of-the-product cycle, high technology and low technology. 

The concept of flexible production is central for understanding all of the industries studied. 
In fact, from the point of view of human capital development and schooling, the key change in the 
economy for both the manufacturing and service sectors is a shift from mass production to 
flexible production. Key to flexible production is the functional flexibility inherent in computer 
software. When production depends on "hard" automation, the retooling required to produce varied 
output is very costly. Under a "hard" technological regime, the objective is long production runs that 
drive down per-unit cost. Ever since Henry Ford mobilized he labor of low-skilled factory wor^ere 
through the assembly line to replace teams of skilled workers, "hard' technology has almost always 
been synonymous with the specialization of labor. 

As technologies become computer-based, they become "flexible" in that retooling simply 
requires rcprogrammiiig, thus allowing shorter production nins and more varied or customized 
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production. Under a flexible production regime, the objtf tive is to combine the customizing implicit 
in craft production at the cos. savings of mass production. Flexibility has usually been achieved by 
reversing Ford's process: moving back up the range of skill levels, shifting from specialized to general 
purpose tools and machines, and reorganizing how people get the work done. 

Let me now briefly describe the results from two case studies: banking and textiles. 

The banking industry has been subject to three forces: 

• increased international competition; 

• increased domestic competition as the result of deregulation; and 

• computerization. 

Before computerization and deregulation, banking involved few services or "products", and its mode of 
operation was a mass production mode — the rapid and accurate processing of millions of a small 
number of different types of transactions. During this era top bank management consisted of college 
graduate generalists; the bank branches operated with a branch manager, assistant manager, head teller, 
tellers, and clerk/typists who did the routine paperwork for activities such as opening accounts. The 
tellers were usually high school graduates with traditional accounting skills, and promotions to low 
level management came from this group. 

In the last decade bank deregulation has generated an explosion of services — from three or 
four to as many as 35, as banks compete for market shares. This explosion drives banks toward a 
market and customer orientation — toward customizing. In other words, it has forced banks out of a 
mass production mode toward a flexible production strategy, with consequent changes in skill 
requirements and staffing patterns. 

Today the teller job is highly routinized, simply a human alternative for customers who do not 
like to use automated banking services. The desk jobs, previously the clerkAypist jobs, are still the 
jobs that deal with customers' service needs. However, individuals in these jobs now must be able to 
analyze a much wider array of the customer's financial needs, understand the array of the bank's 
financial services, and, if possible, produce a match— in other words, make a sale. Banks find that 
they can hire part-time and less-educated help for the highly routinized teller jobs, but must hire 
college graduates for what used to be the clerk-typist jobs. Banks find that they need people who can 



analyze and deal systematically with an array of data. Promotions now come out of the desk jobs, not 
the teller jobs— in fact, tellers are essentially isolated from promotion opportunities in the bank. At 
the same time, the skill requirements at the top of the bank have also changed. Banks now need, not 
college graduate generalists, but highly trained specialists — financial analysts and computer systems 
analysts, for example. 

The textile industry competes on the basis of cost, quality, service, and product choice. 
During most of the post-war era, the U.S. textile industry focused on cost-cutting through the 
rationalization of the production of long runs of fabric. 

However, developments in textile markets— indeed, in markets for almost all goods and 
services— have put limits on the industry's ability to use a mass production strategy. The greater 
segmentation of markets and the faster changing of styles have shrunk the market for large production 
runs of identical fabric. Even such a staple mass-produced commodity as denim now comes in dozens 
of weaves, colors, and finishes. Faster changing seasons have also had their effect. In apparel, styles 
become obsolete much more rapidly. Thus, apparel makers are less likely to order large quantities of 
the same material. The changes in styles are reflected in increases in stock-outs and markdowns. 
Forced maikdowns, which are necessary when retailers fail to sell items during the appropriate season, 
have increased by 50 percent during the last decade. Industry estimates suggest that losses from stock- 
outs, which occur when retailers run out of hot items, amount to 8 percent of sales. 

Among U.S. textile producers it has now become an arricie of faith that the textile industry 
must become more ' market driven", that is, the industry must be capable of producing shorter runs of 
many more styles and must become more flexible. Managers of every mill studied reported increases 
in the number of styles produced-— from three to thirty-five in two years; from one hundred to three 
hundred in five years. 

What has happened to skill requirements in the textile industry? In this industry most jobs are 
machine operator jobs (tower skilled) or machine maintenance jobs (higher skilled). The ratio between 
the two is changing, from 4.2 operators to or* technician in 1975 to 3.5 operators to one technician in 
1985. For the operator jobs, technological innovation means that each particular task is easier. 
However, this narrow conception of skills is misleading; many operator jobs today are more 
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demanding. First, the new textile machines, such as programmable knitters, are much more expensive 
than the equipment they replaced. Operators must now try to prevent machine stoppages — "down- 
tune" is now much more costly. This requires a broader understanding of the production process 
within which the operator works. It is no longer enough for individuals just to understand the 
particular task to which they are assigned. 

Second, because of the increase in the number of styles produced by each mill, ma' operators 
are likely to be engaged in a greater variety of activities and in more of the activities necessary for 
changing styles. Their jobs are less well-defined than they used to be, and the tempo of production 
places a greater burden on operators to function within this uncertainty. As one personnel manager for 
a plant noted, "Our operations change too fast to be able to spell everything out. Operators have to be 
better able to figure things out for themselves." 

Third, textile firms are also becoming more actively involved with working jointly with clients 
in developing new styles and fabrics. So far, at least in the firms visited for this project, this strategy 
does not seem to have had much of an impact on the shop floor, but forward-looking firms are starting 
to consider how the operators could contribute. The same could be said for on-going technological 
innovations. Many of the most important changes have been small adaptations of existing machines, 
and operators could make important contributions to these efforts. 

The higher level positions in the mills also need greater skills and educational preparation than 
they did in the past In the textile industry, the skilled occupations involve machine repair. In the 
past, textile machines were intricate, but the mechanical principles underlying their construction were 
not complicated. How these machines operated could be visually observed, and experience that many 
workers had in fixing their automobiles or farm machinery was relevant to fixing them. Loom fixers 
and mechanics in spinning and knitting mills were almost always promoted from the ranks of machine 
operators. Working around the machines had already given them a feel for what was necessary, and 
the additional training needed to become a fixer was acquired on the job with little or no formal 
instruction. 



This situation has now changed. Most machines now have microprocessors and other 
electronic components, as well as sophisticated sensors. This equipment is well beyond the experience 



that most workers get in homes and on farms. Since important machine components are not visually 
observable, operating the machines does not provide much of a sense of what it takes to repair and 
maintain them. In other words, to understand, diagnose, and fix the new machines, technicians have to 
be able to represent their structures and processes symbolically in their heads. To do this they have to 
be able to follow complicated manuals, diagrams, and updates provided by the manufacturers. 
Literacy requirements have accordingly shot up. The mills can no longer fill many technician slots 
from their traditionally semi-literate operator labor pool. They are adjusting to the problem in 
different ways. Some, reluctant to disrupt their internal promotion patterns, are paying for employees* 
literacy mining. Others are violating these promotion patterns by hiring better-educated labor in 
lateral moves into the technician jobs. This response strands operators, just as tellers who do not 
obtain more education are cut off from the promotion opportunities of the bank. However, whatever 
the mill response, states in which the mills are concentrated, such as the states of North and South 
Carolina, suddenly have mill owners' support for higher quality elementary and secondary education. 

In sum: Service and manufacturing industries are moving from a production-oriented to a 
product-oriented and customer-oriented world, fnm mass production to flexible production. In all of 
the industries studied, Bailey, Noyelle, and other researchers have found that increased competition, 
volatility, and uncertainty in the market have created strong pressures on all levels of the production 
process to be more responsive to changes in tastes and demand— to "customized consumption' 1 
(Noyelle, 1987). 

Although the ability to work on new machines is important, many of the most important 
changes cannot be understood as quantitative. Asking whether the work requires "more" or "less" skill 
inevitably focuses the analysis on limited and often secondary asoects of the transformation underway. 
Productivity gains are coming as much from changing the way that workers work together, their 
orientation towards their work, and the nature of their responsibility for and involvement in the firm's 
changing strategy and orientation towards the market as from applications of new technology. 

While many jobs used to be based on the repetition of a particular set of well-defined tasks, 
jobs now are more likely to demand varied and unpredictable responses to a variety of stimuli and 
information. Employment now involves interaction in constantly changing ways with production 
technology. The spread of micro-electronics and related technologies does not result only in new 



ERLC 



machines that must be mastered, but in a much deeper change in the way production is organized and 
the ways that workers relate to the production process and to each other 

II. ARE WE GETTING WHAT WE NEED FROM OUR SCHOOLS? 

Changes in the nature and structure of work ark! advances in cognitive science argue for 
fundamental changes in K- 12 in what the schools teach, to whom they teach it, when they teach it, and 
how they teach it. 

What Do Students Need to Learn? 

What do industry studies imply about the core skills that students need to learn? Economic 
changes ctrtainly imply the need for good academic skills. Perhaps the most profound educational 
implication of computers in the workplace is that they force a replacement of observational learning 
with learning acquired primarily through symbols, whether verbal or mathematical (e.g., Scribncr 
and Cole, 1973; Bailey, 1988). 

The textile case yielded one example, technicians now having to represent the structures and 
processes of their machines symbolically in their heads. Another example lies in machining. 
Computerized numerical control (CNQ machines radically alter the processes of machinists' set-up, 
control, and operation, replacing manual set-up and control, the skilled hand and skilled eye, with set- 
up by symbolic command. Whereas the machinist working on a traditional machine reads an 
engineer's blueprint arid then manually adjusts dials and levers to set up a particular operation, a 
machinist on the CNC machine reads the blueprint and then creates commands in a programming 
language to govern the machine's operations. What is important about systems such as these is the* 
they depart in significant ways from the traditional systems of knowledge that reflect accumulated 
product on wisdom. They are content-free, formal, closed conceptual systems that have many of the 
characteristics of "school" subjects, such as mathematics or grammar. Individuals who have elected 
traditional machining were usually, while in school, not thought to have to function within such 
systems. Now they do. 

Changes in the economy, especially flexible production and changes in the time frame for 
production, combine to increase the need for higher order cognitive thinking, even for jobs that we 
usually conceive of as lower skill. Time has become an important competitive weapon (Stalk, 1988; 



Bailey, 1989). Companies that can quickly respond to product or service demand gain a competitive 

edge. If the variation in product and service associated with flexible production multiplies the number 

of decisions that must be made, the time element makes it difficult to buck these deci?:°ns up and £ 

back down supervisory lines. Decisions are necessarily having to be made more frequently on the 

shop floor. Thus, work increasingly requires employees both in higher and lower skill jobs to deal 

with the unfamiliar and with discontinuity; to understand the firm's market environment and the 

organizational context in which the job is embedded in order to make decisions that are increasingly • 

being delegated to the shop floor, to understand their technologies well enough to generate initial 

hypotheses about the source of breakdown for maintenance technicians so as to minimize delays in the 

production process. In sum, there is a stunning parallel between the cognitive requirements of today's 

workplace and the defining characteristics of higher order thinking, 2 and this parallel affects workers ^ 

in lower as well as higher skill jobs. 



The forces just described are also flattening out company hierarchies, eliminating supervisory 
and middle management positions. Supervisory functions are being increasingly delegated to the 
worker and/or to the team, requiring of previously supervised workers, not only the ability to make 
decisions previously made by their supervisors, but also the ability to self-regulate or self-direct. 

Changes in the economy imply the need to know how to learn- -in other words, how to 
organize social and technological resources to transform what is unfamiliar into the mastered, a 
process that requires knowing how to identify the limits of one's own knowledge, how to ask germane 
questions, how to penetrate poor documentation, and how to identify sources of information. The 
volatility of markets produces a volatility in job tasks — witness the profound transformation of the 
teller job or the job of operator in textile mills. As Noyelle (1987) observes, M We are moving into an 
era in which the traditional separation between working and learning is disappearing, with learning 
becoming increasingly integrated into a person's work life." (p. 121) 



2 Resnick (1987a) defines higher order cognitive thinking as: being nonalgorithmic — the path of 
action is not fully specified in advance; being complex— the total path is not mentally "visible" from 
any single vantage point; often yielding multiple solutions, each with costs and benefits, rather than 
unique solutions; involving nuanced judgment and interpretation; requiring the application of 
multiple, sometimes conflicting, criteria; involving uncertainty— not everything bearing on the task 
is known; involving self-regulation of the thinking process, not regulation by others; involving 
imposing meaning, finding structure in apparent disorder, and being effortful (p.3). 

8 
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Finally, changes in the economy require teamwork abilities and the ability to resolve 
conflicts. Under mass production, employees, especially those in factory floor and "back office" jobs, 
often worked alone, albeit in physical proximity to each other. As job responsibilities broaden and 
increasingly intermesh, workers have to function collaboratively — and classic research in social 
psychology shows that individual competence does not generalize to team competence. For example, 
pilot error accounts for an increasing percent of fatal airline crashes worldwide, and many analyses 
have pinpointed poor team performance as an important component of that error. 3 

As the labor force becomes increasingly multicultural, and as job content changes rapidly and 
in confusing ways, communication problems also increase between workers, generating the need for 
interpersonal communication and conflict resolution skills. These problems self-evidently reduce 
productivity; more subtly, they interfere with an important social mechanism for learning on the 
job — peer help (Scribner, personal communication). 

Who Should Learn? 

The skills just described are generic in that, in general, they cut across industries and 
occupations. Thus, everyone needs to learn them, not just some people. This does not mean that 
everyone needs to learn them in the same way. It does mean that, for these skills, our educational 
objectives for everyone need to be roughly the same. 

This idea is problematic for the United States, With its traditions of elite and mass education. 
Like other industrialized nations, the United States has harbored two quite distinct educational 
traditions— one concerned with elite education, the other with mass education. As Resnick (1987a) 
points out, these traditions conceived of schooling differently, had different clienteles, and held 
different goals for their students. Thus, for example, in the case of higher order cognitive thinking, 
although "...it is not new to include thinking, problem solving, and reasoning in someone's curriculum, 



3 As a recent New York Times article on cockpit error observed, "Two- and three-man airline 
flight crews.. .often don't work well together." In one example, the article noted a sharply critical 
FA A report on a major airline that had recently experienced several serious near accidents: "There is 
no evidence that Delta crews are (on the whole) either unprofessional or purposefully 
negligent... .Rather.. .crew members are frequently acting as individuals rather than as members of a 
smoothly functioning team." (William Stockton, "Trouble in the Cockpit, 1 ' New York Times 
Magazine, March 27, 1988, pp.38-40, 60, 63, 66-67.) 



it is new to include it in everyone's curriculum." (p,7) This becomes one of the challenges facing 
compulsory American schooling— to make, for example, thinking and problem solving a regular pan 
of a school program for all of the population, even rural populations, even minorities, even the 
poor — to assume that all individuals, not just an elite, can become competent thinkers. 

When Should Th*y Learn? 

Early. We usually think about preparing students for the labor market during high school. 
However, we are talking about generic work-related skills here, not occupationally-specific ones; for 
these, high school is too late. It is implausible to think that high school sophomores educated in a 
passive learning regime for the first nine years of their schooling can learn to self-regulate their 
learning in the tenth year. We can make analogous arguments about learning how to learn, about 
learning how to function effectively in teams, or about learning how to resolve conflicts. 

For exaaiple, the most important single message of modem research on the nature of thinking 

is that the kinds of activities traditionally associated with thinking arc not limited to advanced levels of 

development As Resnick notes, 

M [T]hese activities are an intimate part of even elementary leaming....In fact, the term 
"higher order" skills is probably itself fundamentally misleading, for it suggests that 
another set of skills, presumably called "lower order," needs to come first. This 
assumption ....[ijmplicitly ....justifies long years of drill on the "basics" before thinking 
and problem solving are demanded... .[RJesearch suggests that failure to cultivate 
aspects of [higher order cognitive] thinking may be the source of major learning 
difficulties even in elementary school, (p.8) 

How Should These SkiUs be Taught? 

From the perspective of schooling, the "whats" that should he taught do not necessarily 
translate into jourses. They may have more implications for how material is presented than with what 
is presented. 

This discussion relies heavily on pioneering work in the cognitive sciences on non-school 
learning and its implications for how we structure formal learning. 4 At the heart of this research is 



4 This section relies especiauy on the woric of Sylvia Scribner (e.g., 1974, 1981, 1984, 1986/, 
who helped launch this research direction in the eaily 1970s and now pursues it in the context of a 
research program for the National Center on Education and Employment. 
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the presumption that intelligence and expertise are built out of interaction with the environment, not in 
isolation from it This work implicitly challenges our traditional distinctions between "head" and 
"hand", between "academic" and "vocational" education, between "education" and 'training", and 
between school-based aM work-based learning. 

In a brilliant synthesis of the woric in this field, Lauren Resnick (1987b) delineates four broad 
contrasts between in-school and out-of-school mental activity that raise profound questions about the 
utility and effectiveness of schooling for non-school activity, including work of all types. They 
stimulate us to rethink — radically reUvnk — hew we teach in school. 

The first contrast is between individual cognition in school versus shared cognition outside. 
Although group activities occur in school, students are ultimately judged on what they can do by 
themselves. M* 4 ch of the core activity of the school— homework or in-class exercises— is designed as 
individual work. For the most part, students fail or succeed at a task independently of what other 
students do (aside from grading on a curve). By contrast, a great deal of activity outside of school is 
socially shared: work, personal life, and recreation take ptece in social systems in which what one 
person is able to do depends fundamentally on what others do and in wnich "successful" functioning 
depends upon the mesh of several individuals' mental and physical performances. This contrast argues 
for much more team and co-operative learning, the student being held accountable for both individual 
and team performance. 

The second contrast is between pure mentation in school versus tool manipulation. In 
school, the greatest premium is placed on "pure thought" activities— what individuals can do without 
dependence on "external crutches"— whether books and notes, calculators, or other complex 
instruments. While some of these tools may be used, even encouraged, during "learning", they are 
almost always absent during tests of performance. Thus, school becomes an institution that values 
thought that is independent of the physical and cognitive tools that are a vital and de< ring part of 
virtually all practical activity. Out of school, by contrast, most mental activities are intimately 
involved with and shaped by the physical and intellectual tools available, and the criteria for 
competence include the expert use of tools. 
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This contrast suggests that student performance should be judged relative to the student's 
abilities to make effective use of tools, not independent of them. 

The third contrast is between symbol manipulation in school versus reasoning about things 
and situations that make sense to peo|* j outside of school. School learning is mostly symbol- 
based, to such an extent that connections to the things being symbolized are often lost. Outside of 
school, actions are intimately connected with things and events, and because one is engaged with 
things and situations that make sense to people, people do not fall into the trap of forgetting what their 
calculations or their reasoning is about Their mental activities make sense in terms of their immediate 
effects, and their actions are grounded in the logic of immediate situations. In school, however, there 
is a very large tendency for symbolic activities to become detached from any meaningful context 
School learning then becomes a matter of learning rules and saying or writing things according to the 
rules. This focus on symbols detached from thei r referents can create difficulties even for school 
learning itself. For example, it can lead to systematic and persistent arithmetic errors of a kind that 
seem virtually absent in practical arithmetic. 

This contrast between in-school and out-of-school mental activity suggests that school-based 
learning is not strongly related to out-of-school activity for any individual. However, the disjuncture 
between school and non-school settings would seem to be particularly detrimental for at-risk 
learner*. 5 We frequently assume that at-risk and not-at-risk populations differ in how they learn most 
effectively. Although it is an empirical question, variations in learning performances may attest partly 
to individual differences in the willingness to tolerate or in the ability to make sense out of a school- 
based or school-1 ke experience that is relatively isolated from non-school experience. 

This contrast, then, suggests instruction in the context of what makes sense to people. 
Especially for the academically less inclined, schools try to introduce "things and situations that make 
sense to people" either by putting the student in vocational education or by linking schooling to 



5 I define at-risk learners as those who do not perform well in traditional schools or training 
programs arranged like traditional schools, either because they are not very good at standard academic 
subjects or— and this is an exceedingly important "or"— because they do not want to be or do not see 
the point of being good at them. It is important to note that although at-risk learners come 
disproportionately from poor families, almost every family either has a child of this sort or friends 
with a child of this sort. 

12 
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outside jobs, as in organized part-time woric and part-time school ventures. There can be good reasons 
for doing either of these things, but not as a way of compensating for the often impoverished learning 
contexts of academic courses. 

If earlier I argued that all students, not just the academically inclined, need to master higher 
order cognitive skills, now I am arguing that all students, not just the academically disinclined, need 
contextualized learning. I am talking about "vitalizing," not "vocationalizing," schooling. I think that 
working out what this means, how to do it, understanding how individual schools and teachers are 
already approximating it, represents one of the most exciting challenges ahead of us. 

We are not starting from ground zero. The best teachers in our best schools already 
instinctively "vitalize 11 even the most "academic" of subjects. A fifth grade teacher in a McLean, 
Virginia school runs a simulation of a small economy in the classroom to give her students experience 
with fundamental economic concepts such as competition, monopolies, bankruptcy, rents, or taxation. 
Reading a discussion of markets, sellers, and competition in a textbook means much more to the 
student who, just the previous day, waged a price war with a seatmate to comer the market on hot dog 
sales. "Taxation" means much more when another seatmate who represented government has bought 
the classroom door, forcing everyone to pay taxes every time they need to go in or out of the 
room — for example, to get water to boil their hot dogs. 

Our vocational-technical, theme, or magnet schools hold important clues — many of these 
explicitly organize around meaningful activities that real people do in the real world, such as business 
careers, the performing arts, or science. 

Another clue lies in the new information technologies. As the Office of Technology 

Assessment (1988) notes: 

[With the new information technologies, it] may soon be possible to manipulate 
images and sounds as easily as the printed word. This means being able to break 
some of the barriers of abstraction that separate "scholarship" from the world people 
see, hear, and understand.... 

These innovations can radically change 'he performance t.id structure of the 
educational system. The new generation of technologies.. .are qualitatively different 
from the film strips, television shows, and other techniques that have been used in 
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limited ways to augment instruction in the past. They represent something 
fundamentally new....(pp-242-245) 

Many of us, myself included, glaze when we heai about "new information technologies." We have 
lived through generations of "silver bullet" computer-based instructional programs that whizzed by, 
only to bury themselves in the wall. \ v e have seen the numbing, pedestrian ways in which the schools 
man? fi to use computers. However, I think that the OTA is right this time about the potential. The 
problem is getting from here to there. A sustained investment will be required to develop these 
technologies for pedagogic purposes. A major restructuring of the schools will be required to integrate 
these technologies into schooling. The information technologies do not parachute successfully into any 
organization, whether firm or school. Their effective integration into the organization always requires 
organizational restructuring. 



The fourth contrast is between generalized learning in school versus situation-specific 
competencies outside. In school we aim for general, widely usable skills and theoretical principles. # 
Indeed, the major claim for school-type instruction is, usually, its generality and power of transfer. 
Yet outside, to be truly skillful, people must develop situation-specific forms of competence. The 
"packages" of knowledge and skill that schools provide seem unlikely to map directly onto the clusters 
of knowledge that students will actually use in their work. This seems true even for highly technical * 
knowledge, where schooling is intended to provide direct professional training. Studies of expert 
radiologists, electronic trouble-shooters, and lawyers all re sal a surprising lack of transfer of 
theoretical principles, processes, or skills learned in school to professional practice. For example, 
Morris and Rouse (1985) found that extensive training in electronics and troubleshooting theories 
provided very little knowledge and fewer skills directly applicable to performing electronic 
troubleshooting. All of this points toward the possibility that very little can be transported directly 
from school to out-of-school use. Both the structure of the knowledge used and the social structure of f 
its use may be more fundamentally mismatched than we had previously thought. 

This contrast argues for posing problems to students that train them to adapt general 
knowledge to specific situations. How we do this presents us all with a challenge. Resnick notes that • 
situation-specific learning is limiting. Studies have shown that wnen the situation is changed from the 
familiar— for example, by asking bookies in Brazil to accept unusual bets that cannot b s constructed 
from their tables (Schliemann and Acioly, in press)— unschooled individuals have a great deal of 
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difficulty and may fail entirely. Schooled people do better, although— and this is an important 
point— they rarely use the supposedly general algorithms that they have been taught in school and 
instead invent new solutions specifically appropriate to the situation at hand. However, what do they 
invent? How and why? Expert radiologists interpret X-rays using mental processes different from 
those taught in medical courses, textbooks, and even hospital teaching rounds (Lesgold, et al., in 
press). What processes do experts use? How did they learn them? Can they be taught? How? Do 
they bridge the worlds of the specific and the general? These studies raise questions that seem at the 
heart of the dilemma for schools. 6 

m. HOW DO WE GET WHAT WE NEED FROM OUR SCHOOLS? 

I suggest that U.S. schooling, as it usually occurs, is failing, that a symptom and cause of this 
failure is the disjunction between schools and non-school settings, that our accountability assessments 
reflect this disjuncture, and that rethought assessments are one engine for restructuring the educational 
enterprise. 

If assessment results are public, comparative, and affect the reputations of schools, teachers, 
principals, and school graduates, we find th? f they affect curriculum and pedagogy. What and how we 
measure gets taught; what and how we do not measure does not get taught Assessment, then, means 
danger and opportunity. 

The issue is whether these assessments are designed to tell all parties with a stake in our 
educational system what they want or need to know in a way that they can understand. I submit that 



6 Scribner's work centers on this problem. In studies of workers in jobs that vary substantially in 
their symbolic and physical requirements, she found a set of attributes common to expert performance 
in all of the jobs: (1) flexibility in modes of solution to formally identical problems; (2) creative 
shortcuts that simplify and economize on mental and physical effort ("least effort" basis for organizing 
work); and (3) fine-tuning to the environment and effective utilization of setting-specific knowledge. 
She is now trying to understand how experts create expertise — for example, how less-skilled workers 
learn manufacturing resource planning (MRP) systems, a computer integrated information system 
analogous to the closed, logical systems of school, such as grammar or mathematics. Less-skilled 
workers often did not handle closed systems well when they were in school. How do they learn die 
MRP? What facilitates the learning? Do the processes of moving from novice to expert in the plant 
have implications for teaching closed systems in school? 
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the answer here is "no." Current assessments are both a symptom and a cause of the disjunction 
between schools and non-school settings. 

Today assessment almost always consists of paper and pencil, norm-referenced, multiple 
choice tests, the items being secret and selected to reflect what schools and text books define as 
important, and the act of test-taking being screened and private. 

Educators increasingly appreciate the authenticity problems with standard assessment 
Fundamentally, paper and pencil, multiple choice tests do not model the performance demands or 
resources of non-school settings. For example, multiple choice tests preclude displays of problem- 
solving and other metacognitive skills, implicitly presuming that "competence" is the ability to retrieve 
the "right" facts from a warehouse of facts. They thus encourage routine drill in bits and pieces and 
militate against the development of higher order cognitive thinking, which has more to do with the 
intelligent use of judgment than with "right" answers. 

Since multiple choice tests stress "know thats", we also cannot see whether students "know 
how" or how they integrate know thats and know hows into their performance. Since they are paper 
and pencil tests, we cannot see if students can use the other tools and resources that are routinely 
available in and whose use is critical to performance in non-school settings. 

If we increasingly understand the authenticity problems with standard assessment, the elements 
and consequences of secrecy in assessment are not understood. These elements are curious and 
profoundly ramifying in their consequences. 

One dimension of secrecy is the literal secrecy of the test items and of the test-taking process. 
Multiple choice tests "sample" "know thats" from the domain that test-takers are expected to have 
mastered. Thus, protecting the validity of these tests entails secrecy of items and of the test-taking 
process. If test takers know in advance what they will be asked on the test or have access to others' 
knowledge during the test taking process, we cannot use the test-taker's performance on items sampled 
from the domain to infer his or her command of the domain itself. 
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The fundamental problem is the multiple choice nature of the tests, the secrecy entailed by 
them dimply underscoring the authenticity problems that a multiple choice paradigm poses. Mastery 
outside of school only has meaning in the context of known and practiced activities. Thus, if we want 
to assess student mastery, secrecy about the competencies that the student will be asked to display 
makes no sen:*. In designing authentic outcom: measures, key ideas are "demonstration" or 
,, performance , \ Good models here are how we judge artistic performance — for example, the piano or 
violin recital, or athletic performances in Olympic tryouts, or demonstrations of competence that boy 
scouts and girl scouts go through to win merit badges. The idea of performance presumes knowing in 
advance. In fact, in many instances of performance-based assessment, the performer not only knows 
in advance what he or she will be asked to perform, but is allowed to choose that performance — the 
musical piece, dance, or gymnastic display — that he or she thinks best showcases his or her 
capabilities. 

If we want to assess, not student mastery, but how students handle new or unencountered 
situations, we want to see how well they frame the problem and identify and organize 
resources — books, experts, other tools— to help them solve it. In this case the secrecy of the test- 
taking process makes little sense. It cuts off access to those resources which, intelligently used, define 
the effective learner and problem-solver in the real world. 

Secret tests and testing also interfere with assessments as signals. Because the items are secret 
and the tests taken in privacy, our assessments do not serve to signal our performance expectations for 
students or help them see what competent performance looks like. 

Assessments are secret in a second, more subtle, and deeply troubling way — in the sense of 
"obscure", "opaque", and "inaccessible." Because the items assess competence at what schools and 
text books define as important, these tests and their results cany no intuitive meaning to anyone 
outside of the world of educators and educational test designers. They measure performances that 
have meaning to educators and within the context of testing theory. They rarely measure how well 
our students can perform activities that have meaning in the non-school world and ^at, by virtue of 
having meaning in that world, are transparently understood and credible to student parents, 
employers, politicians, the media, and the larger public. They thus fail to measure objectives that 
parties with interests in the outcomes of our educational system can understand, "see", and debate. 
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Similarly, when assessments are norm-referenced, we ki.ow how individuals perform relative 
to the performance distribution of the population against which the test is benchmarked. However, we 
do not know how they perform relative to some criterion or performance standard that is socially 
valued and understood. Again, only educators, test developers, and sophisticated corporate trainers 
know what the results mean. 

I submit that the consequences of opaque assessments are profound. They do not operate as 
signals that co-ordinate the decisions of teaci'ers, schools administrators, parents, students, employers, 
and the polity. They simply mirror and echo the disjunctive world of education. As such, they cannot 
help equilibrate skill demands and skill formation. They cannot support a broad communal 
understanding of how the schools are performing. They cannot become a vehicle for communicating 
and building a dialogue between the groups that have to work together if we are to restructure 
American education. 

ASSESSMENTS AS CLEAR SIGNALS 

I suggest that redesigning our assessments to eliminate secrecy and opaqueness can help to 
bring the outcomes of school and the skill and knowledge needs of non-school settings more into line 
with one another. The key is assessment as a clear signalling system — specifically, assessment that: 

• signals to schools and students the performance expectations that prevail in non-school 
settings, such as the workplace. 

• serves to instruct students in what the adult community defines as competent 
and weak performance. 

• signals how well schools are preparing students for performances that are 
socially valued and meaningful. 

• becomes a vehicle for communicating and building a dialogue between groups 
that have to co-ordinate their activities to restructure education — groups such 
as educators, parents, students, political leaders, employers, the media, and the 
larger public. 

Assessments that function a*> a clear signalling system require openness, in the sense of being non- 
secret and transparently understandable. This requires: (1) that what is assessed and the testing process 
are public, (2) that what is assessed are performances meaningful and valued in non-school settings, 
and (3) that assessment results are reported relative to performance standards that have meaning and 
are valued in non-school settings. 
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